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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address ~ 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

Responsive to communication(s) filed on the amendments of 3/7. 3/22 and 3/29 . 
2a)D This action is FINAL. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) (3 Claim(s) 1-1 16. 1 18-129. 131-166. 168-179. 181-192. 195-204 and 207-251 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) |g| Claim(s) 22.24-39 A1. 44-11 6. 11 8-1 29. 131-166. 168-179. 181-192. 195-204.207-236 and 238-251 is/are allowed. 

6) I3 Claim(s) 1-5.11-13. 15-21.23.40.42.43 and 237 is/are rejected. 

7) [3 Claim(s) 6-10 and 14 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) O The drawing(s) filed on is/are: a)D accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

1 1) D The proposed drawing correction filed on ; is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) S Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (0- 

a)l3 All b)Q Some*c)D None of: 

1 . £3 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* see the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) Q The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 

Attachment(s) 

1) ^ Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) Paper No(s). . 

2) S Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) □ Notice of Informal Patent Application (PTO-152) 

3) Q Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) □ Other: 
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DETAILED ACTION 
Allowable Subject Matter 

1. Claims 22, 24-39, 41, 42/22, 43/22, 44-54, 56-63, 65-116, 118-129, 131-165 and 240- 
243 are still allowed. 

2. Claims 55, 64, 166, 168-179, 181-192, 195-204, 207-236, 238-239 and 244-251 are 
newly allowed. Claims 55, 64, 238 and 239 are allowed because no prior art anticipates or 
suggests comparing first transformed image data and second transformed image data in order to 
distinguish transformed image data relating to the object from transformed image data relating to 
its shadow. Claims 166, 177, 179, 187, 244 and 245 are allowed because no prior art anticipates 
or suggests generating reliability information indicating the reliability of the image data in 
dependence upon the angle between the user-selected viewing direction from which the input 
image was recorded. Claims 168-176 are allowed because they depend, directly or indirectly, on 
claim 166; claims 181-186 are allowed because they depend, directly or indirectly, on claim 179. 
Claims 178 and 188 are allowed because no prior art anticipates or suggests an indicator of a 
quality of the generated image data is produced for output to the user in dependence upon the 
angle between the user-selected viewing direction and the viewing direction of the camera. 
Claims 191, 203, 246 and 249-251 are allowed because no prior art anticipates or suggests 
identifying the cameras having a viewing direction within a predetermined angle of the user- 
selected viewing direction as identified cameras. Claims 192 and 195-200 are allowed because 
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they depend, directly or indirectly, on claim 191; claims 204 and 207-209 are allowed because 
they depend, directly or indirectly, on claim 203. Claims 201-202 and 210-211 are allowed 
because no prior art anticipates or suggests a plurality of cameras related to image data quality, 
wherein the plurality of camera parameters are tested in a predetermined order, with the selection 
of image data being made once the tests identify a camera parameter which is sufficiently different 
for the cameras. Claims 214, 223-226, 232-234 and 247-248 are allowed because no prior art 
anticipates or suggests testing whether first and second images of the object displayed from the 
generated image data will be discontinuous by testing whether the image data for the object in the 
second image in the sequence differs by more than a predetermined amount from predetermined 
image data. Claims 215-222 are allowed because they depend, directly or indirectly, on claim 
214; claims 227-23 1 are allowed because they depend, directly or indirectly, on claim 226. Claim 
238 is allowed because no prior art anticipates or suggests an image data transformer for applying 
a transformation to the identified image data from the first camera which defines a mapping from 
the ground plane in the space of the image data of the first camera to a surface in a modelling 
space, and for applying a transformation to the identified image data from the second camera 
which defines a mapping from the ground plane in the space of the image data of the second 
camera to the surface in the modelling space. 

3. Claims 6-10 and 14 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the base 
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claim and any intervening claims. 

4. The following is a statement of reasons for the indication of allowable subject matter: No 
prior art anticipates or suggests, when the heights of the corresponding representations are not 
within the predetermined amount of each other, splitting the taller representation into a first 
portion having a height corresponding to the height of the smaller representation and a second 
portion comprising the remaining part of the taller representation, and wherein the further 
representation is defined by re-positioning the second portion in the three-dimensional model 
(claim 6); and determining the width of the planar surface by the width of the bounding polygon in 
the image data, and calculating the height of the planar surface using the aspect ratio of the 
bounding polygon in the image data (claim 14). Claims 7-10 are allowable because they depend 
on claim 6. 

5. However, the claims below, allowed in the last Office action are, upon review, rejected. 

Claim Rejections - 35 USC §103 

6. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness 

rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art 
are such that the subject matter as a whole would have been obvious at the time the invention was made to 
a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be 
negatived by the manner in which the invention was made. 
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7. Claims 1-5, 11-13, 15,21,23,40,42/1,42/21,43/1,43/21 and 237 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Marks (U.S. Pat. No. 6,025,847) in view of Grumet 
(U.S.Pat. No. 4,601,053). 

8. With respect to claims 1, 21, 23, 40 and 237, Marks, in disclosing a 3D modeling system 
with visual feedback, also discloses a method of processing image data defining a plurality of 
sequences of images, each from a respective camera, of a plurality of objects moving in a scene to 
produce signals defining representations of the object in a 3D computer model, and generating 
object representations in the three-dimensional computer model (col. 6, 11.42-48). 

9. However, Marks does not disclose the specific details of the applicants' method; these are 
disclosed by Grumet. 

10. Grumet, in disclosing an automatic TV ranging system, also discloses: 

• processing image data from first and second cameras to identify image data relating to 
objects in the scene (col. 3, 11.22-42); 

• processing the identified image data (on a TV monitor screen) from the first camera 
(camera 10, FIG. la) for each object to define an object representation in a modelling 
space having a height (range Ri, FIG. la) dependent upon the image data for the object 
from the first camera; 

• processing the identified image data (on a TV monitor screen) from the second camera 
(camera 12, FIG. la) for each object to define an object representation in a modelling 
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space having a height (range R 2? FIG. la) dependent upon the image data for the object 
from the second camera; and 
• comparing the height of the representation of each object generated in dependence upon 
image data from the first camera with the height of the representation of the corresponding 
object generated in dependence upon image data from the second camera (compare range 
Ri to range R2 in FIG. 1 a). 

1 1 . Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to use the Grumet system of rangefinding in the Marks modeling system. 
This would provide the logic required to process a complete 3D scene to identify and store x, y z 
or other coordinates for each object (Grumet, col. 1, 11.30-33). 

12. The other claims in this rejection will now be considered: With respect to claim 2, Marks 
discloses the modelling space in which the object representations are defined using image data 
from the first camera and image data from the second camera as three-dimensional computer 
model (col.6, 11.42-48). 

13. Concerning claim 3, Marks discloses (col.2, 11.5-17) the step of generating object 
representations in the three-dimensional computer model comprises modifying the taller 
representation (col.2, 11.12-14) when the heights of corresponding representations are not within a 
predetermined amount (actual location of physical object, col.2, 11.9-10) of each other. 
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14. Regarding claim 4, Marks discloses a method wherein when the heights of corresponding 
representations are not within the predetermined amount of each other, the taller representation is 
modified to give a representation having a height based on the height of the smaller representation 
(col.2, 11.5-17). 

15. With respect to claim 5, Marks discloses a method wherein when the heights of 
corresponding representations are not within the predetermined amount of each other, a further 
representation is defined in the three-dimensional model using part of the image data from which 
the taller representation was defined (col.2, 11.5-1 1). 

16. Concerning claim 11, Marks discloses an object representation defined as a planar surface 
(col.2, 11. 1 1-14). Neither Marks nor Grumet directly disclose a base on a predetermined surface in 
the modelling space and with a position and size in dependence upon a polygon bounding the 
image data for the object, but it is obvious from Grumet that a TV screen shows objects, which 
are bounded by polygons, in their real-life proportions to each other. 

17. Regarding claim 12, Grumet discloses the polygon as a rectangle (a square TV screen is a 
rectangle). 

18. With respect to claim 13, Grumet discloses the sides of the rectangle as being parallel to 
the sides of the image (in a rectangular TV screen, the sides are parallel). 
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19. Concerning claim 15, Grumet discloses a planar surface lying within a vertical plane 
(whatever is shown in a Grumet TV screen lies in a vertical plane because the TV screen has 
dimensions on the y (vertical) axis). 

20. Regarding claim 42/1 and 42/21, Marks discloses a storage medium for storing 
instructions for causing a programmable processing apparatus to become operable to perform a 
method according to claim 1 (memory device 104, FIG.l). 

21. Finally, with respect to claim 43/1 and 43/21, Marks discloses a signal conveying 
instructions for causing a programmable processing apparatus to become operable to perform a 
method according to claim 1 (computer 100, FIG.l). 

22. Therefore, in view of the foregoing, claims 1-5, 1 1-13, 15, 21, 23, 40, 42/1, 42/21, 43/1, 
43/21 and 237 are rejected under 35 U.S.C. 103(a) as being unpatentable over Marks in view of 
Grumet. 

23. Claims 16-20 are rejected under 35 U.S.C. 103(a) as being unpatentable over Marks in 
view of Grumet and further in view of Goldberg et al. ("Goldberg, 55 U.S. Pat. No. 5,877,779). 

24. However, neither Marks nor Grumet disclose, with respect to claim 16, generating image 
data by rendering an image of the three-dimensional computer model in which texture data based 
on the processed image data is rendered onto the representation of each object. This is disclosed 
by the Goldberg method and apparatus for efficient rendering of 3D scenes at col. 2, 11.5-1 1. 
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25. Therefore, it would have been obvious to one of ordinary skill in the art at the time this 
invention was made to incorporate the Goldberg rendering method in the Marks-Grumet 
modeling system. This would greatly increase the efficiency and throughput of graphics data in 
the rendering pipeline (Goldberg, Abstract, next to the last sentence). 

26. The remainder of the claims in this rejection will now be considered: With respect to claim 
17, Marks discloses the step of generating a signal conveying the image data in col. 6, 11.42-48. 

27. Concerning claim 18 and 20, Marks discloses the step of recording the signal (col 2, 11.5-8- 
creating the 3D model for modification implies storing ("recording")). 

28. Finally, regarding claim 19, Marks discloses the step of displaying an image of the objects 
using the generated image data (col. 2, 11. 11-14). 

29. Therefore, in view of the foregoing, claims 16-20 are rejected under 35 U.S. C. 103(a) as 
being unpatentable over Marks in view of Grumet and further in view of Goldberg. 

Claim Rejections - 35 USC § 101 

30. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. 

The claimed invention is directed to non-statutory subject matter. Claims 43/1 and 43/21 are 
rejected. 
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3 1 . Patentable subject matter is held to exclude "laws of nature, natural phenomena, and 
abstract ideas". Diamondv. Diehr, 450 U.S. 175, 185, 101 S.Ct 1048, 1056 (1981). The 
applicants' signal claims 43/1 and 43/21 disclose an abstract idea (a signal) which is not tangibly 
employed on a computer readable medium. Only an applicant's claims are entitled to the 
protection of the patent system; therefore claims, if expressing ideas in a mathematical form, must 
describe something beyond the manipulation of ideas in order to qualify as patentable subject 
matter. In re Warmerdam, at 1360. Given the absence of any practical effect or significant 
independent physical acts, claims 43/1 and 43/21 fail to adequately define the claimed invention 
within the domain of patentable subject matter. 

Remarks 

32. The examiner agreed with all of the applicants' arguments in response to the last Office 
action and was persuaded that the proffered amendments to the mentioned claims made the claims 
allowable. However, a fresh look at the claims rejected above revealed that these claims were not 
allowable. Since at least claim 1 was not amended between the last Office action and this one, the 
rejections in this Office action are non-final. 

Conclusion 

33. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Lance Sealey whose telephone number is (703) 305-0026. The 
examiner can normally be reached Monday-Friday from 7:00 am to 3:30 pm EDT. 
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34. If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mark Zimmerman, can be reached on (703) 305-9798. The fax phone number for the 
organization where this application or proceeding is assigned is (703) 872-93 14. 

35. Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305-4700 or 
the Customer Service Office at (703) 306-0377. 




MARK ZIMMERMAN 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2600 



